Reading a single, well-organized text narrative is a common means of conveying information. It is also common to combine images with that text to convey information more effectively to the reader. Many books, encyclopedias, and on-line documents, primarily in nature, contain images illustrating the topic of discussion, and conversely, some form of text accompanies the images.
In the printed medium, references from the narrative to the image are expressed within the text, and references from the image to the narrative are expressed as annotations to the image. Consequently, a heavy burden is placed on the reader to redirect his attention from one element (image or narrative) to the other and back again. This slows the process of absorbing and understanding the information.
Further, the printed medium does not provide a way within a single image to animate that image, for example, changes in the image over time, different aspects of the image, or alternative versions of the image. Printed materials therefore present separate images for each temporal instance, aspect, or variance of the image. This wastes space and provides less information to the reader.
Computer-based media, particularly the Internet, deliver increasingly large amounts of information. A sizable portion of this information concerns topics that are best presented using a blend of text and image or graphical media. Prior art methods for presenting such information involve the same approach as described above for printed media. A second prior art method, specific to computer-based media in general and the Internet, places “hyperlinks” within the text or image. When activated, the hyperlinks “jump” the reader to a related text or image. However, this second method suffers from three substantial problems. First, it is usually impossible to jump to a particular element of a graphic or image. Second, jumping to text is at the page or at best paragraph level. Third, the jump itself interrupts the continuity of information delivery to the reader, then interrupts again by forcing the reader to jump a second time to return to the original information.
Computers have long supported the display of animated images. Internet browsers also support animation through the inclusion of “add-on” or “plug-in” software such as MACROMEDIA FLASH (computer software for creating animation, sound, graphics, presentations, and illustrations; computer software for viewing and playing animation, sound, graphics, presentations, and illustrations; computer software for use in authoring for the web; computer software for enhancing the capabilities of web browser software), or by coding custom applications for embedding in a web page which are typically written using the JAVA (computer programs for use in developing and executing other computer programs on computers, computer networks, and global communications networks, and instruction manuals sold therewith; computer programs for use in navigating, browsing, transferring information, and distributing and viewing other computer programs on computers, computer networks and global communications networks) computer language. The requirement of specialized browser add-ons is inconvenient, and in certain environments, totally inapplicable. More important, these approaches do not provide the ability to finely synchronize individual image elements with text elements, thus denying the reader an integrated information experience.
News organizations publish interactive maps and diagrams on the Internet. In this type of map or diagram the text and image information do not interact, and no single coherent narrative is available for reading or listening. “Headline” or “pop-up text” approaches replace the single narrative approach with snippets and bullets of text, but do not cohere to create a story. For instance, there is no narrative that can be read aloud to a sight-impaired user using text-to-voice technology.
The Synchronized Multimedia Integration Language (SMIL, pronounced “smile”) claims to enable simple authoring of “rich media”/multimedia presentations that integrate streaming audio and video with images, text or any other media type. SMIL is a language, not a method or device. SMIL synchronizes in time, the image, sound, and text components of a presentation. SMIL, although supporting text, treats it primarily as “headlines” to be flashed at the reader during the viewing of images. SMIL does not provide the precise detail needed for the interrelationship between text and image required for certain types of applications.
An Internet consortium initiative known as Structured Vector Graphics (SVG) provides two approaches for animating or interacting with the graphics it defines. In the first approach, statements within the language itself define the animation that is to occur, such as with the <animate> element. The second is to manipulate the SVG objects by means of a scripting language. Although this permits a variety of effects and applications, SVG is primarily a graphics definition language and by itself does not provide the fine detail of interrelationship between text and image required for certain types of applications.
Apple Computer has developed a user-assistance system called “AppleGuide”. The objective of AppleGuide is to provide the user with information on how to use his computer, similar to a “help system”. AppleGuide displays a narrative text; the user then accesses a portion of that text and is informed of specific actions to take by a fat red circle around the relevant location on the computer screen. This system solves the problem of linking elements of text to elements of image, but only for the limited purpose where the image is that of an actual computer screen displaying user commands. Also, this system does not provide support in the “reverse” direction, from the relevant image to the corresponding portion of the narrative.
A standard is currently emerging for electronic books or “e-books” and is used primarily for reading e-books on portable viewers. This standard defines the structure for e-books and an XML-based language for encoding them. It also permits graphics to be defined as part of the book by using typical web-based graphics encoding such as GIF and PNG. However, it does
not provide any ability to interrelate text and graphics other than the normal web-based hyperlink.
An example of information that involves both text and image is a cooking recipe. A traditional recipe lists the ingredients, followed by a sequence of steps describing how to prepare, combine, or cook the ingredients. Images are limited to pictures of the finished dish, or diagrams of how to perform individual steps. The reader must transform the text into a mental model to carry out the preparation of the recipe, usually repeatedly consulting the text in order to verify or refresh the information. The efficiency of conveying the information in the recipe to the human brain is thus very low. Some prior art exists using flowcharts to graphically depict steps in food preparation. This approach is deficient in that it fails to use easily understandable visual icons, does not allow the presentation of multiple paths of preparation, and is focused on food safety issues. Cooking recipes can be presented effectively with interrelated narrative text and images.
Tourism information is another example of the use of both text and image. A plethora of material is available to tourists regarding places of interest to visit while traveling. The majority of these materials are in the form of printed books, although information is also available on the Internet and other computer-based media. The prior art of placing text next to images has been employed for conveying travel information. Historically information was presented on the printed page and more recently on the Internet, yet these methods of presentation are inefficient. For instance, the narrative may describe a particular walking tour of Seville, discussing various sites on the tour while referring to an accompanying map. The reader must constantly refer back and forth between the map and the text, looking on the map for sites mentioned in the narrative, or looking within the narrative for the description of sites on the map. Tourism information can be presented more effectively with interrelated narrative text and images.
Weather reports are an excellent example of information that involves both narrative and image. The image in this case is the weather map. Conceptually, a location mentioned in the narrative interlinks to the location shown on the map. The same applies to meteorological features such as weather fronts. Printed media, such as newspapers, present a narrative weather report accompanied by a map, however, newspapers fail to provide the reader with any assistance in linking the narrative report with the map. Weather information conveyed over the Internet typically gives an on-line narrative report accompanied by a map, which often includes animation showing the movement of storms and weather fronts over a period of time. Here again, text and images are treated as separate, static objects, forcing the reader to continually bounce back and forth between text and image in order to digest the information.
A huge amount of material has been published regarding games, including card and board games such as chess, bridge, and “igo”, an Asian strategy board game. These materials are a prime example of information having a visual (image) and a verbal (textual or narrative) component.
Typically such materials employ images of the board together with text to provide information about the sequence of moves, or in the case of card games, cards that have been played. The sequence of moves actually played in a game, or variations that might have been played, are displayed along with a commentary on the sequence. These materials also show board positions representing problems for the player to solve along with the successful and unsuccessful solutions to those problems. Historically, such materials were published in books and magazines. A major weakness of the print medium for board game information is that board positions can only be shown statically. Any sequence of moves must be shown by numbers or letters on the board, with moves then referred to in the accompanying narrative using those numbers or letters.
A number of specialized computer applications are available which allow a sequence of moves in a board game, as well as variations, to be shown in animated fashion on a computer. Often these applications take as input files in the “SGF” format for the game “igo”, and the “PBN” (Portable Bridge Notation) format for chess and bridge, which were developed to encode move sequence and variation information in computer-readable form. Some applications also display short commentaries on any move or position for which a commentary is available in the SGF or other file. Classic examples are SmartGo, an SGF editor and viewer and, more recently CGoban2, a SGF editor and viewer that functions as a client for users playing each other via the Kiseido Go Server. However, these applications are deficient as solutions to the problem of computer-based board game information display in that they are separate applications, must be created separately for each computer platform, and may require complex installation and configuration procedures. They are also primarily oriented to board display and do not represent a useful delivery mechanism for text-oriented commentary on board game positions, moves, or sequence.
Other examples of prior art for presentation of board game information within the internet browser environment include a typical JAVA applet-based board game (applet is a program written in the JAVA programming language that can be included in an HTML page, much in the same way an image is included) display program which has many of the same defects mentioned above for specialized computer applications; namely, the “add-on” or “plug-in” must be available for the hardware, operating system, and the browser used. It must then be downloaded, configured, and maintained by the end-user. In the case of an embedded JAVA application, again, the end-user's computer must have the appropriate JAVA environment installed, and the application (“applet”) must be downloaded to the user's computer. Further, the applet permits the display of board game figures only, requiring the use of HTML to provide the commentary. Again, this limits the two-way integration between figure and commentary that is crucial to the seamless presentation of board game material on-line. All these restrictions severely limit the application of these solutions.
At the same time, some igo and other board game commentary has been made available for computer-based consumption primarily using the HTML format used by Internet browsers. Providing igo commentary in HTML format has an advantage in that no specialized software is required to be downloaded, installed, configured, maintained, or updated by the end user as it can be viewed using virtually any Internet browser. However, in the prior art we find no use of the scripting language-based programmability of Internet browsers in order to provide animated go diagrams, or diagrams tied to any narrative discussion. Instead, the images are inevitably static, with the same defects inherent in using printed media.
The features and advantages of the present invention will be presented in more detail in the following specification of the invention and the figures.